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ABSTRACT 

The need for study programs that fit better to the 
characteristics of learners, their interests, and the labor market 
has led education from being supply driven toward being demand 
driven. The construction of individual study programs requires 
appropriate assessment methods. The technologies generally available 
for assessment seem to reinforce the supply-driven nature of 
assessment, but demand-driven education, where students influence 
content selection and teaching is growing. The central finding of 
research of the past 15 years is that the key to developing an 
integrated and generative knowledge base is to build on the learner's 
prior knowledge. An empirical study is reported that focuses on the 
application of new forms of assessment for individual study programs, 
concentrating on prior knowledge and progress assessment. 
Questionnaire responses of 2,000 university students show that 
students largely agree with the use of prior knowledge state and 
progress tests and that their aims are improvement of knowledge and 
of study methods. Seven graphs and one table present study findings. 
(Contains 60 references.) (SLD) 
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Abstract 



The need for study programmes that fit better to the characteristics of the learner, his interests and the 
changes in the labour market has become a subject for researchers in recent years. This evolution of 
education from being supply driven towards being demand driven needs to be understood in the larger 
context of societal, economic, political, scientific and technological developments. From an educational 
point of view, the construction of individual study programmes needs appropriate assessment methods for 
students. 

The paper focuses generally on the predominance of supply driven education. The small set of available 
* technologies' seems to reinforce the nature of supply driven education. Supply driven education is 
pressurised changes in the societal context. Educational theories proclaim new ideas, but in general 
educational practice reflects predominantly supply driven features. The same seems to be true of the 
emerging lT-applications. During the last decade, demand driven education, where students influence 
content selections and didactical elaborations, gets growing attention. 

In a first part, the paper focuses at the theoretical background of the problem of appropriate assessment 
methods for such flexible study programmes. The central finding of investigations of the past fifteen years is 
that a key to developing an integrated and generative knowledge base is to build upon the learner's prior 
knowledge. Glaser and De Corte state: "Indeed, new learning is exceedingly difficult when prior informal as 
well as formal knowledge is not used as a springboard for future learning. It has also become more and 
more obvious, that in contrast to the traditional measures of aptitude, the assessment of prior knowledge and 
skill is not only a much more precise predictor of learning, but provides in addition a more useful basis for 
instruction and guidance" (in: Dochy, 1992). Past research has shown that prior knowledge has a main 
influence on the learning of students and is a better predictor for study success than other variables. Also 
the variables 'study progress' and 'knowledge gain 1 has shown to be useful predictors for study success 
(Imbos, 1991; Tan & Imbos, 1993). Moreover it has become clear that the use of assessment techniques as 
instruments for gathering information on study progress and for directing study guidance should be 
increased. However, implementing such research results in educational practice is not that easy. 

In a second part, an empirical study is reported. This study is focussing a first step in the application of new 
forms of assessment for individual study programmes, i.e. prior knowledge state and progress assessment. 
This first step is a clear answer to the question if students themselves do support the 'overall assessment 
prophecy' and if and how they do intend clearly to use the instruments. The present study investigates this 
matter, i.e. students' perceptions, using a questionnaire for a sample of 2000 university students. 
The results show that students largely agree with the prophecy and show a high preparedness to use the 
proposed assessment instruments, being prior knowledge state tests and progress tests. Remarkable is the 
finding that student do want to use the instruments for strongly different purposes (knowing their strenghts 
and weaknesses; directing their study guidance; attaining the fastest study progress as possible within their 
study programmes) and that there are differences between specific domains and between beginning and 
advanced students. 
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1 Introduction and theoretical background 

1.1 Earlier research on prior knowledge and progress 

Earlier research (Dochy, 1992; Dochy & Valcke r 1993; Dochy & Alexander, 1994) shows that prior knowledge 
of students is a determining factor in the learning process of students in higher education, both in a regular 
university as in a distance teaching university. A methodology for using Prior Knowledge State Tests was 
introduced. This methodology took as a starting point the finding that the structure is one of the most important 
aspects in taking the students' prior knowledge into account. 

Starting from the overall-assessment prophecy, i.e. that assessment becomes an integral part of instruction, we 
huve investigated the usefulness of introducing tests for assessing the prior knowledge state of students at the 
beginning of a their study. A similar procedure for assessing the students' knowledge states during the study 
period has been called progress tests (Dochy & Bouwens, 1992; Dochy, 1992). 

1.2 A conceptual framework for studying knowledge acquisition and the structure of knowledge 

Starting from a European- American conceptual framework (Dochy & Alexander, 1994), the prior knowledge 
state (PKS) is defined as the knowledge state comprising existing declarative and procedural knowledge which 
meets the following conditions: it is present before the implementation of a particular learning task, it is 
available or able to be recalled or reconstructed; it is relevant for the achievement of the objectives of the 
learning task, it is organized in structured schemata, it is to a certain degree transferable or applicable to other 
learning tasks, it is dynamic in nature. For a clear definition of all terms used in relation to knowledge 
acquisition we refer to Dochy and Alexander (1994). 

The structure of knowledge issue has been dealt with by a variety of theoretical disciplr is: cognitive 
psychology, educational psychology, artificial intelligence, etc. From a pragmatic point of viev the issue has 
also been of prime importance in applied sciences like instructional psychology, curriculum development theories 
and psychometrics. An early, cognitive-theoretical approach appears in the work of Ausubel (1968) who argues 
that new knowledge is only acquired to the extent that it is meaningfully related to existing knowledge. Ausubel 
maintains that knowledge is organized primarily in a hierarchical fashion, which implies that mastery of higher 
knowledge levels assumes mastery of all lower knowledge levels. Additionally, Ausubel advances that the 
various pieces of information integrated within a particular knowledge structure are highly interrelated. Thus, 
the more structured prior knowledge, the more flexible and easy the acquisition of new knowledge becomes. 

Ausubel's conceptualization of learning as the assimilation into prior knowledge, is echoed and extended in 
Mayer's (1979) schema theory. New knowledge is - according to Mayer - assimilated into a hierarchy of 
progressive'/ more specific content within the learner's knowledge base. Thus, the basic learning process can 
be described as the assimilation of new knowledge within hierarchically ordered schemata. 

Another benchmark is set by the 'elaboration theory'. According to this theory, multiple access avenues 
become available to the learner by the activation of alternate relational paths. This theory also depended on the 
assumption that knowledge acquisition is facilitated to the extent that information is organized in a hierarchically 
integrated mode (Reigeluth & Stein, 1983). 

From a rather pragmatic, e.g. instructional-psychological, point of view the structure-of-knowledge paradigm 
needs to be investigated in detail in order to find more efficient ways for using instructional technology. 
Research into ways of utilizing knowledge states (Dochy, 1992) indicate e.g. that the different components of 
the PKS should be taken into account (e.g. at the subject-matter level) and that components of the PKS along 
other dimensions (e.g. educational-psychological dimension) could be helpful in educational settings for 
diagnosis and as a basis for intervention. 

An important conclusion from our former research indicates that it looks promising to analyze in more detail 
the complex of components of the Knowledge States (KS). A first attempt in this direction focused on the 
structure of the PKS along the content dimension (Dochy, 1992). Note that we use the term 'component* to 
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refer to these subparts of the prior knowledge state that relate to a specific parameter along a knowledge profile 
dimension, as our knowledge state tests did refer to at the content dimension. 

In order to clearly understand this statement, it is necessary to stress that three important concepts are 
introduced in the former paragraph: the KS, components of the KS and a complex of components. These three 
concepts refer to the value attached to the specific 'structure' of the KS. Our earlier research revealed that such 
a structure could be indicated in prior knowledge, e.g. along the content dimension. We did for instance 
discriminate between the optimal requisite prior knowledge state and the mathematics prior knowledge state. 
But it was also suggested that the differentiation of components of the KS along other dimensions could be 
helpful to diagnose and support educational practice. For a clear distinction between 'parameter' and 'dimension' 
we refer to Figure 1 . 



1.3 Knowledge profiles 

As such, the concept of 'knowledge profiles' is not found in literature. Only 'student profiles' (Wolf, et al., 
1991) and 'cognitive profile' (Letteri et al., 1982) have some similarity in meaning. 

Our use of the concept 'profile' is derived from the practice, common in educational research, of plotting as 
a graph (or profile) the raw or standardized scores of a group or individual on certain parameters (Keeves, 
1988). In analyzing research data, comparisons are made between persons or groups of persons in terms of a 
set of measurements about related aspects. For each person or group a profile can be obtained by combining 
the results of the set of parameters. The comparison between individual profiles is known by the generic term 
'profile analysis'. 




Figure 1: Example of a knowledge profile 

Figure 1 shows the relationships between some key concepts. A 'dimension' is the basis to construct a 
knowledge profile. Each dimension represents an approach towards the structure of knowledge since it 
introduces a related set of KS components. These components are named parameters. 

From an instructional-psychological point of view, knowledge profiles can give practical indications of student 
achievement and lean in^, thereby making it possible to direct the learning process. In a recent overview of 
student assessment, W M et al. (1991) advocated this approach. According to these author' there is a need for 
new educational psych c>ivietrics in line with the much changed perceptions of educational achievement. The 
educational world must come to terms with new premises, multiple paths towards the prior knowledge state, 
more developmentally-oriented assessments and the students entering school with widely varying backgrounds. 
For our part, we take account of these changes by trying to identify the components of the prior knowledge 
state, by implementing prior knowledge state tests and by initialisation of plans to use these tests several times 
a year to monitor students' progress. 
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In this context it is necessary to come to an agreement on the relevant parameters for describing student 
performance and it is critical to develop ways of looking at 'student profiles*: " unless we develop these kinds 
of differentiated portraits of student performance within a domain, it is difficult to envision student assessment 
ever informing, rather than merely measuring, the educational process" (Wolf et ah, 1991). 

Our multi-profile approach generates 'profile analysis* (Dochy & Valcke, 1993; Va'cke & Dochy, 1991), an 
application of multivariate analysis of variance (MANOV A) in which several dependent variables are measured 
on the same scale (Tabachnick & Fidell, 1989). This profile analysis can provide relevant information with 
diagnostic and remedial value. In this way, the profiles help us to identify learning deficits to be remediated. 

Data were found that support the relevance and validity of the knowledge profile dimensions. Two approaches 
were to be adopted, based on data gathered during an investigation involving a large sample of university 
students. First, we analyzed the extent to which the parameters along the dimensions give information about 
different components of the PKS. Second, we analyzed the discriminatory power of the knowledge profile 
dimensions to make apparent PKS differences between a variety of student sub-populations (Wagemans et al., 
1991; Valcke & Dochy, 1991; Dochy & Valcke, 1993). 
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2 The scope of the present investigation 



The starting point of this investigation has been a variety of problems observed in the context of higher 
education, e.g. the problem of equal opportunities for all students at the start, the problem of multifunctionality 
of modules in modular education, the problem of sequence in which course modules are studied, the problem 
of differing students, the problem of making appropriate use of prior knowledge. Prior knowledge is a central 
issue in relation to each of these specific topics. 

To solve these problems, it is not enough to work out a methodology and practical instruments. Our new 
approach towards the analysis of prior knowledge foresees that in situations where there are significant 
differences between the PKS of specific subpopulations, the profile dimensions will be helpful to detect and 
dissect in more detail the strengths and weaknesses of the students involved. This might be a promising starting 
point for differentiated diagnostic and guidance approaches. 

Next to that, a clear view on how students perceive these new instruments and on their future habits will be 
necessary: their needs, their preparedness to use, their possible investment, etc. This will be investigated in the 
present study. A sufficient acceptance for the overall assessment prophecy and its practical implementation will 
urge to a new teaching-learning model as presented in Figure 2. 

Then instruction is to be reconsidered, at least in open learning environments, on the basis of a new educational 
model of the learning process in which the overall assessment takes a central place and the students* prior 
knowledge state is the starting point. 



Student 



T est.ng 



Pr i o r * Knowledge 
State test 



Progress 
tests 



F i na 



Know I edge base 



goa I s 



Lear n i ng activities 



Learni ng 


activity 


1 










M 


Learn i ng 


activity 


2 


t= 
D 








Lear n i ng 


activity 


3 


i 








A 


Learn i ng 


activity 


n 





Figure 2: Assessment at the core of the teaching-learning model 
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According to this model, the student starts with stating his learning goals. These relate to a certain part of the 
knowledge base (the content or the whole of a university's courses)(arrow A). After having taken a prior 
knowledge state test, the learning goals are reformulated (if necessary) and the students start with the 
appropriate learning tasks (arrow B). During the leaning process the student takes progress tests regularly to 
check his progress, to determine the required guidance and to identify the subsequent learning tasks (arrow C). 

2.1 Underlying for theory driven assumptions 
2.1.1 Effects of prior knowledge 

Earlier cited theories of Ausubel, Mayer, and Reigeluth & Stein are important since they stress the 'structured', 
hierarchical nature of prior knowledge and indicate that prior knowledge is an important variable in learning. 
Additional support, especially for the importance of prior knowledge in learning (measured by a Prior 
Knowledge State Test), both in the sense of high correlations and highly explained variance (20 to 45% in 
ecologically valid settings and up to 90% in experimental settings), has been provided by the investigations of 
Bloom (1976), Lodewijks (1981), Parkerson et al. (1984), Korkel (1987), Alexander et al. (1989), Weinert 
(1989), Minnaert and Janssen (1992), Dochy (1992) and Valcke and Dochy (1991). From these investigations, 
one can derive that the 'building' comparison is important in learning, which means that the student's knowledge 
state is not only important at the start of his study, but remains an important variable during the study (i.e. at 
the start of each building block, at the l>eginning of each learning task, etc.). 

Although - as shown earlier - the impact of prior knowledge is often stressed to this extent that all learning 
might even depend on it (Resnick, 1981^, also other factors like other student characteristics do influence the 
learning process and can interact with the impact of prior knowledge. But, it is not yet clear which personal or 
contextual variables play a significant role in this context (Ferguson-Hessler, 1989). Differentiating between 
groups of students on the basis of such variables does not seem to make much sense for educational practice. 
Moreover, our research indicates that, if different variables are taken into account, prior knowledge always has 
the strongest general effect and overrules other variables in descriptive and explanatory models (see also 
Ethington, 1990; Bruinsma & Geurts, 1988; Dochy, 1988). Analysis in terms of the High Knowledge (HK) and 
the Low Knowledge (LK) groups (in the sense of higher education and duration of work experience) show a 
clear distinction between groups (Dochy & Steenbakkers, 1988). The majority of the HK group students found 
the course easy, whereas the opinions of the LK group students were divided. This assumption supports the 
general results of research into prior knowledge and the fundamental issue of the different Prior Knowledge 
theories, i.e. that students' prior knowledge makes learning easier. 

This can be explained by the influence of the direct effect of prior knowledge on learning. By direct effect is 
■\kderstood all effects as found in situations in which prior knowledge was found to perform an influence on 
study results of students. In those studies, the amount of knowledge at which the student has the disposal of, 
is directly related to the expected study success (see Niestadt & Dochy, 1992). Certainly there are investigations 
which contradict the general direct effect. The study by Palmer et al. (1979) is such an example. However, they 
conclude that having high school economics may in some cases mislead them into over-confidence. 
Besides studies into direct effects of prior knowledge on study results, research has been done into the indirect 
effects on study results, for example effects on study time. Indirect effects come to surface when investigating 
for example interactions between prior knowledge and instruction. Prior knowledge is here found to have effects 
on study results that are not plainly caused by the knowledge possessed. In those studies, prior knowledge is 
found to affect the required time for study, the effort one has to spare, etcetera. When we draw our attention 
to the indirect effect of prior knowledge, then all consequences of prior knowledge are paramount, in which 
prior knowledge appears to be of influence for effects which, in turn, directly affect a student's study results. 
For example, Saunders (1980) found that students who had taken one or more domain-specific high school 
courses knew significantly more at the end of a university principles course, even though they reported working 
significantly fewer hours on the university course. Other researchers (see Niestadt & Dochy, 1992) concluded 
students with high SAT scores learned more per time unit in any type of intellectual endeavour. A conclusion 
from these studies into indirect effects can be that instruction affects new achievements as it interacts with the 
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learner's prior knowledge state or his familiarity with the subject matter. Tobias (1976) described a range of 
levels of instructional support and embedded support devices which can minimize individual differences between 
students. This proposal seems to find a reasonable basis in educational research. The well-known study of 
Bonello (1984) was already a clear warning to education based on self-study in the sense that in providing 
courses to students with different prior knowledge levels, one should take account of these differences in the 
individuals 1 materials. 

The facilitating effect of prior knowledge is generally recognized by educational researchers as being the most 
important positive effect on learning. In Dochy (1992) different theories that give an explanation for this finding 
are explained. Nevertheless, we should not forget that not all facilitating effects are the 'direct* results of prior 
knowledge. For the purpose of this review, we can make a distinction between: 

a. a direct effect of prior knowledge facilitating the learning process and leading to better study results; 

b. an indirect effect of prior knowledge, optimizing the clarity of study materials; and 

c. an indirect effect of prior knowledge, optimizing the use of instructional and learning time. 

Further, given the fact that we will look for differences between domains in the present study, it is important 
to stress that it has been convincingly shown that prior knowledge states of students are highly domain-specific, 
assuming that students will learn more effectively in some domains than in others (Alexander, 1992; Alexander 
& Kulikowich, 1991; Dochy, 1992). 

2.1.2 Students' perceptions of their own prior knowledge 

An additional approach to determine the prior knowledge level of the students, is to ask the students themselves 
to estimate their mastery level. In an experiment by Lodewijks (1981), it was shown that prior knowledge could 
account for 36% of the explained variance in post-test scores. In this instance prior knowledge was considered 
to be the subjective rating of familiarity with the content of the learning task. However, when considering the 
Falchikov and Boud (1989) meta-analysis of 48 student self-assessment studies, the researchers came to more 
differentiated conclusions: most studies found positive effect sizes, indicating overrating on the part of the 
students. It appears that more experienced students in a particular field (that is, students with more prior 
knowledge) are more accurate estimators, whereas students taking introductory courses appear to make 
particularly inaccurate self-assessments. 

In our own study (Wagemans, Valcke & Dochy, 1991), it was shown that the estimation of PKS level through 
self-assessment by the students is not very reliable in comparison with objective prior knowledge state tests. The 
estimations do not reflect the levels of PKS as measured by the (two different) objective tests (Figure 3 and 4). 
For our purpose, it can be concluded that self-assessment is not useful, at least not for introductory courses, 
as also shown by Falchikov and Boud (1989). Moreover, it was found that students have very differing 
conceptions of 'prior knowledge', which makes a self-rating procedure invalid (Dochy, 1992). 
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Figure 3: Differences between self-assessment levels and real PKS levels (PKST1) 
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Figure 4: Differences between self-assessment levels and real PKS levels (PKST2) 
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2.2 Assessment procedures 
2.2.1 Final examinations 

Several assessment procedures have recently been operationalised at our university in a distance education 
context. Next to the traditional administration of paper-and-pencil examinations in study centres (including 
multiple-choice format to open-ended questions) on the occasion of three moments a year, the SYS-system is 
used. In the SYS-system, the computer generates a unique examination for each individual student by selecting 
items from an item-bank. This automated evaluation format fits extremely well in the Open university 
philosophy guaranteeing a high degree of openness and flexibility. The student can take a final examination any 
moment he wants to. The necessity to inform the study centre about the intent to take an examination two weeks 
beforehand is the only time-limitation. After solving the questions and problems, the student gets feedback about 
his final score. This score is confirmed about two weeks later after item-analysis and test-analysis. 
Information is available about appreciation and the perception of students of the SYS-evaluation format. In a 
1991 -study among 1 123 Open university students, attitudes towards the SYS-evaluation system were as follows: 
43% said they consider the system to be moderately suited, 50% indicated the system was well suited. In a 
comparable 1992-study among Ou-students, the appreciation of this examination format raises up to 62% of the 
students (1992-II). When asked to choose among three examination formats, students favoured the most flexible 
format (SYS-examination). Two consecutive studies revealed the following proportions: 

Appreciation of examination formats 
1991 and 1992 



% stuaenta 




tradHionai»3 K0ditionfti>3 SYS 

Examination formats 

■■ 1091 1902 



Figure 5: Appreciation of examination formats 

The data in figure 5 undoubtedly indicate the growing appreciation of the SYS-system. This obvious 
preference for SYS-examinations does not differ between students studying different domains (1991-11 and 
1992-1). Further analysis of the quantitative results reveal that students especially appreciate the SYS-format 
because it allows to adapt the examination-moment to the personal study plan. 

Apart from evaluative data about the SYS-examination system, there is hardly Information available about 
the student opinions in relation to the more traditional examination procedures or the examination content, 
the types of items, etc. in general. Moreover, and related to the topic of this research, we observe that 
nearly all investments in evaluation has been in relation to 'final examinations/tests'. Prior knowledge state 
tests and/or progress tests only recently have been considered as possible topics for a substantial investment 
in educational research and development. 
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2.2.2 Alternative assessment procedures: prior knowledge and progress tests 

Prior knowledge state tests (PKST) related to the content of a course have showed to be highly reliable 
instruments (for economics and mathematics, see Dochy, 1992; for history, see Elen, 1993) with a high 
predictive validity as stated earlier. In his profound investigation into prescription for instructional design, 
Elen (1993) concludes that from his different design parameters (cognitive skills, metacognitive knowledge 
and skills, and prior knowledge), only "prior knowledge has been shown to significantly affect learning 
outcomes and is to a certain extent also of determining importance". According to Elen, the value of prior 
knowledge cannot be underestimated. 

Further, from our experiences at the university of Maastricht, a university with a problem-based curriculum 
using progress tests four times a year for all students, it was clear that using tests several times a year to 
monitor students* progress was a very useful and informative source, both for students and for professors 
(Dochy 8c Wijnen, 1987). In a study concerning the use of progress tests, Imbos (1989) concluded that 
progress tests are very useful, even for first year students. He discriminated between * first year curriculum 
items* and *general items* and found that laymen have a chance for success of .14 on first year items and 
are able to answer one third of the questions. Imbos suggests two concrete aspects for amelioration of 
progress tests: (1) ordering the items based on p-values and (2) giving students who score vary low a 
tailored test (selected sample of the same itembank adapted to his prior knowledge level). 
Shavelson, Baxter and Pine (1992) have been investigating science performance for several years, designing 
tests to foster good teaching and learning as well as to monitor student progress. Shavelson et al. (1994) 
were using alternative assessment technologies, consistent wkh the emerging constructivism assumptions 
about learning. One can say that these assumptions are surely correct for our target group, being adult 
students. Interesting feedback on student progress was supported by scoring the assessments so as to capture 
not just the right answer, but also the reasonableness of the procedure that was used to solve the problem 
(Shavelson, 1994). 

Finally, the results of the 1992 study (Dochy, 1992) showed that there was a need for more formative 
evaluation during the study periods. These stipulations have led to work out so-called *progress tests*, 
consisting of items at final examination level. In this way, the progress tests give information on the 
discrepancy the students have to bridge in order to reach the final goals of the course(s). 

Related to the overall-assessment prophecy (Dochy, 1992) and me use of progress tests, clearly an 
innovation can lead to interesting results. Snow (1990) states that new approaches to cognitive assessment in 
education should be directed towards initial states, daily skill and strategy transitions, weekly knowledge 
transitions or monthly progress in a course. Clearly, our earlier mentioned model of progress tests fits with 
Snow*s last approach. There seems to be no doubt that such an innovative model is useful when looking at 
empirical research. Powell et al. (1990) comes to the following conclusions. Probably lei>s than 10 percent 
of the students demonstrated sufficient progress to complete a multi-course program within a reasonable 
length of time. Moreover, 60 percent of first-time students fail to earn any credits within 30 months of 
being admitted. It seems clear that the problem cannot be handled by contracting students because nearly 
half of the students are unable (or perhaps unwilling at the moment) to state their study program intentions 
(Powe\ et al., 1990). This fits with the results of van der Velden et al. (1992), finding that only 16 percent 
of the students in distance education are interested in the advantages of such a study contract. 

2.2.3 Assessment as an alternative support procedure 

As staled above, other - formal - alternative examination provisions are not available in our university. But 
some 'informal* alternative formats, rather known as support devices, are present. For instance, the distance 
education study materials (mostly text-based) comprise a self-test and types of *embedded support devices* 
(ESD) of an evaluative nature. A student can - but is not obliged to - make use of these text-embedded 
provisions and can read the written feedback at the end of each learning unit. 
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In a recent study - with a small sample of law students - the actual use of such embedded evaluative support 
provisions has been investigated (Poelmans et al f 1993). The findings show that questions, prequestions, 
tasks and feedback on tasks were used by at least 80% of the students. Moreover, 77% of the students and 
in most cases up to 90% of the students, made a 'deep lever use of these provisions. Deep level use implies 
that the students extensively pay attention to the ESD, read them and react on them, let the ESD influence 
their study behaviour, etc. In terms of 'appreciation 4 , the evaluative embedded support devices were part of 
a set of highest appreciated ESD. 

In another 1992-study with Ou-students the evaluative support devices in the instructional materials were 
researched with a larger student population from different study domains (N= 1907). This study revealed 
that 84% of the students solve more than 50% of the embedded questions and tasks (Boon et al., 1992). 

These results suggest that students have a need for alternative assessment procedures that fit into their study 
approach, study plan, etc. Also it is suggested that the use of these assessment procedures is depending on 
the accessibility and easiness for daily use of these instruments. But study approaches differ and study plans 
represent a wide variety of strategies used by students. So, despite a rather 'general' demand for alternative 
assessment procedures, there remains the question about the adaptation of the alternative procedures to each 
students' study approach. About this topic, there is hardly information available in the context of distance 
education at the Dutch Open university. In this text we therefore pay much attention to such alternative 
evaluation procedures but link this immediately to the issue of student 'types'. 



2.3 Students' views on innovative procedures 

Concerning the students' views on our stated problem, a preliminary investigation was earned out by Dochy 
and Steenbakkers (1988) to search for an answer to a number of questions on the views of students about 
prior knowledge and the importance that prior knowledge has on learning. 

In-depth telephone interviews were used and a sample of 120 Open University students were approached as 
respondents, divided among fields such as Economics, Natural Sciences and Social Sciences. This sort of 
interview uses pre-structured questions with open response options and can be regarded as a qualitative 
method of research, by Patton (1980) also as 'the standardized open-ended interview'. 
Students were asked if they found the courses that they took related to their work / hobby / experience. Of 
the total research population one third of the students agreed 'hat this was the case, although another third of 
the students thought there was no relation. In this last group there was clearly a majority of the so-called 
low-knowledge group (being students with only a few years work experience and no higher education 
degree). 

Concerning the manner in which new knowledge can best be acquired, students still assign a dominant role 
ix> the transfer of knowledge to the school. When this is broken down into the HK and the LK groups, it 
appears that traditional views of the acquisition of knowledge are strongest among those with the most 
formal education. The LK group believes that it can also derive knowledge from self study and experience. 
Further, it was found that virtually every student thought that the effect of not having the 'prior knowledge' 
would be that the time required for the course takes longer. Another effect would be that where there was 
sufficient motivation and time, the course could be passed, but it would affect the course results i.e., a 
lower score would be achieved in the examination and it would be easier to fail. 

These conclusions are consistent with the results of a study by Powell et al. (1991), investigating success 
and persistence at two distance teaching universities using questionnaires. Powell et al. found that the 
educational qualifications on entry students of both universities were associated with success on the first 
course taken, with students having no post-secondary education performing worse than those with higher 
qualifications. 

The study showed that pass rates of first-time students at both universities were remarkably similar at 36 
percent. Pass rates for students who subsequently registered for more courses increased considerably as 
students progressed through their studies. Moreover, it was found that success on the first course taken is 
highly correlated with re-registration. Only 15 % of students who do not pass their initial course, return to 
take another. 
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2.4 



'Types' of students : a rationale 



As stated above, in this study about prior knowledge state tests and progress tests, much attention is being 
paid to 'types' of students. This interest in 'types 1 of students is an important topic in the general context of 
distance education: courses are to be multi-functional; 'clients' have very different backgrounds/profiles 
(educational level, professional background, study plan, study speed,...); etc. 

The fact we concentrate so much on 'types' of students is the result of empirical, theoretical and pragmatic 
considerations: 

Institutional research with Ou-students (Boon et al, 1990) indicated e.g. the importance of variables 
such as educational background to explain differences in attitudes towards aspects of distance 
education. Our own earlier research in relation to prior knowledge research revealed that prior 
knowledge among students differs depending on their prior educational experiences, but only when 
we group students in two groups, being students with and without an university or higher vocational 
education degree (Dochy, 1992). Research in relation to the actual use of study materials reflect the 
impact of educational level, gender and study plan (Poelmans et al., 1993). 
At the theoretical level, student variables (independent variables), are incorporated in descriptive 
and explanatory theories or model about learning and teaching at the Open university (e.g. Valcke, 
et al., 1992; Vermunt, 1992). 

From a pragmatic point of view, taking account of student variables als answers a growing need to 
reach more efficient and effective education. There is for instance an urgent demand for more cost- 
effective planning and implementation of support and evaluative provisions; e.g. the number of 
study centres and study centre staff, the number of examinations taken by students,.... A key 
variable in this discussion is the 'type* of student for whom support and evaluative provisions in 
study centres are relevant and needed. The importance attached to the development of better intake- 
procedures is but one outcome of this general concern. 

In this study we operational ise 'types' of students by making use of three variables : 

The level of previous educational experiences; we distinguish two groups of students, a group 
having higher education (higher vocational or university degree) and a group of students which 
highest previous educational experience is of secondary school level. 

The level of success in studying in the distance education setting of the Dutch Open university; 
here we distinguish between four groups of students based on the number of certificates: 0, 1-2, 3- 
4, and more than 4 certificates. 

The content domain according to the students* study plan; the Dutch open university distinguishes 
between the following 7 content domains: law, economics, business, technical, natural, cultural and 
social sciences. 

One can question whether the three operational definitions of the variable 'student type* presented here are 
relevant and helpful to explain the way students react to evaluation formats. In our study, we do not neglect 
this question. More specific, we make use of log-linear analysis techniques to evaluate the impact of the 
independent variables when looking for grounds that account for the variance in student responses. 



Six clusters of hypotheses can be put forward taking account of the information obtained from the 
questionnaire (see appendix). The first four sets are related to prior knowledge state tests. The fifth is 
related to progress tests and the sixth set to study contracts: 



3. 



Research Design 
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Research hypotheses 
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a 



e 



b 



c 



d 



f 



Students have no clear view of their prior knowledge level. 
Students do not need prior knowledge state tests. 

The perceived and expected effects of PKST on study variables are limited. 
The perceived effect of PKST on guidance, support is limited. 
Progress tests do not answer specific student needs. 
Study contracts do not answer specific student needs. 



In the results section, each cluster hypothesis will be considered by relating the cluster to specific 
questionnaire items. In this way in-depth results are obtained which are helpful to draw detailed conclusions. 
The reader will notice that section f on 'study contracts* cannot be considered as an alternative evaluation 
format. As will be explained in section 4.3, we did incorporate this topic due to a animated discussion 
actually found at the Dutch Open university. We consider 'study contracts 1 , in the context of this text, as a 
logical extension to the alternative evaluation formats put forward. In this way we want to know whether 
students are willing to attach formal consequences to the results of new evaluation formats. As indicated in 
section 2.2, the impact of the independent variable 'student type 1 will be researched in relation to each 
cluster of hypotheses. 

3.2 Research Instrument 

This investigation was designed in cooperation with the department for market research of our university. 
As part of their yearly research with a sample of students, the authors constructed a specific set of questions 
to be incorporated in a 17-page questionnaire (cf. appendix 1). The set of questions about prior knowledge 
state testing and progress tests made up one section in this questionnaire. 

The items in the questionnaire mirror a specific set of research variables put forward about alternative 
assessment procedures. Each item can be consistently linked to such a variable. The construction of these 
items was not an evident task as will be outlined in the next paragraph. 

If students have no prior experiences with the phenomena researched, how is it possible to get reliable and 
valid responses to question about these phenomena? The designing of a research instrument to get relevant 
information on opinions of students about the alternative assessment procedures researched here., is based on 
the assumption that students are not familiar with these alternative evaluation formats. Therefore the design 
of the instrument takes into account the pitfalls given by Neijens (1987) that can appear in the construction 
of an instrument for collecting informed opinions of a population. According to Neijens (1987) the four 
main indicators for the quality of the evaluated opinions are: direction errors, consistency errors, halo 
effects and judgement variation. Most of these can be avoided by providing correct information about the 
phenomena (and linking to their own experience) and by selecting carefully the presentation of the 
information. 

First, in our design process, this solution was operational ised mirroring the alternative evaluation formats to 
aspects of the actual study process and study behaviour of the students. Secondly, in some cases several 
items were partly overlapping in order to control consistency. Thirdly, by gradually presenting to the 
students other aspects of the alternative evaluative procedures, in terms of inferences on aspects of their 
actual study behaviour and study process, more reliable information can be obtained about their 
appreciation, attitudes, acceptance of the new ideas. In this respect it is to be expected that students will 
show some so-called 'orientation behaviour' during the first questions. Example: Time planning of the study 
process towards the final examination is a normal aspect of student study behaviour. If we show to the 
stiK>r,cs that progress testing can affect their normal time management, students can take positions on a 
better informed base. This example suggests that in constructing the questionnaire care has been taken to 
present to the students other specific implications, consequences rnd characteristics of the alternative 
evaluation formats. Another example is more complex. When testing e.g. the hypothesis "Students do not 
need prior knowledge state tests.", five items in the questionnaire were presented to the students, each of 
them presenting aspects that affect the study behaviour and study process: 

- uncertainty about one mastery when starting a Ou-course 

- time management 
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- the way information obtained from prior knowledge profiles is to be used 

- the need for personalised and written feedback 

- the lack of feedback in current evaluation practices. 

Despite the care taken in constructing the questionnaire, caution will still be needed when interpreting the 
research and when drawing conclusions. The data obtained from the students can still be biased by lack of 
information about and experience with the alternative evaluation formats. 

As stated above, the questionnaire was carefully constructed by gradually adding specificity to the 
implications of alternative evaluation formats in terms of student behaviour and the study process: 

The questionnaire started with a descriptive text giving a short outline of 'prior knowledge' and 

'prior knowledge state tests'. 

Next, questions were put forward about the students' perception, estimation of their personal prior 
knowledge state/level. 

After this, general questions were put forward about their willingness to take prior knowledge state 
tests. 

In the next questionnaire item, students could react to statements about possible uses that could be 
made with t e results of the tests after deriving strong and weak points in one's prior knowledge 
state. 

The following set of questions repeated one by one possible uses of the prior knowledge state test 

results. Here, students could respond in a more elaborated way (5-point scale). Moreover, if 

relevant, an extra variable was added in these questions about the person to make 'use' of the 

information obtained: the student him/herself or advisors/experts. 
The subsequent part of the questionnaire consisted of: 

a short introduction on progress tests 

and two questions about this evaluation format 
The questionnaire ended with: 

a short explanation about study contracts 

and questions about the willingness of students to consider taking study contracts. 
Also the questionnaire contained two questions about: 

- feedback on the tests that are embedded in the course materials 

- and the opinion of students about the feedback value of the actual Ou-exarranations. 

A first version of the questionnaire was screened by a team of educationalists and adapted following their 
comments. A second version was screened by the staff of the "Onderzoek & Evaluatie" department of the 
Dutch Open university. After revision, the final version was drawn up and incorporated in the larger 
questionnaire. 

3.3 Research Population 

An a-select sample of 2000 Ou-students was taken from the student population that subscribed for a course 
in the period September 1990 - September 1991 (N = 35.313). This sample was stratified according to the 
variable 'success' in passing course modules. Of this sample, 1 159 students responded by returning the 
questionnaire (58%). A weighting of the respondents was necessary taking the variables 'number of courses 
taken' and 'number of modules passed'. Every course exists of a certain number of modules depending on 
the study load of the course. Therefore, the modules give a more precise indication of a students' 
studyprogress. The results of the weighting procedure was a weighted number of 1907 students, each having 
passed the same amount of modules in a certain domain. As a matter of fact, in the following paragraphs n 
will always refer to this weighted number. 

3.4 Procedure and data analysis 

As stated above, the questions in relation to the alternative examination formats were part of a larger 
questionnaire. Early in 1992, the questionnaire was sent to the students. The extraction of the research data 
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from the questionnaire forms was done by staff of the department for market research. Further analysis was 
executed by means of a descriptive analysis of the answers on the questionnaire, a Chi-square analysis and 
log-linear analyses (table 1). 

The Chi-square analysis of two-dimensional tables was executed to evaluate the independency of each 
specific student type variable from the large set of dependent variables as measured by specific (sets of) 
items in the questionnaire (1. prior educational level, 2. the degree of success in studying at a distance and 
3. the content domain in which the students study). When we report in this .text 'significant differences*, this 
implies that the independence hypothesis is to be rejected. Ia reporting, we will then especially focus on 
those cells in the table that are particularly illustrative to base our conclusions in relation to the hypotheses 
put forward. 

Next a log-linear analysis of the potential combined impact of two student type variables (previous 
educational experience and study success) was done to explain the data in the three-dimensional table (main 
effects and interaction effects); the analysis focused on a test of independence of the three variables in the 
analysis and took account of the first two student type variables as independent variables and the specific 
answer on a question as the dependent variable; 

Finally, a hierarchical log-linear analysis explored what kind of model could best fit and significantly 
explain the data in the three-dimensional table. This final model is expected to be the simplest model 
explaining still the highest proportion of variance. The results of the analysis will only be used if the final 
model is different from the saturated model. 



[Insert here table 1] 
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4 Research results 

In this section, the results are discussed following the six clusters of hypotheses as discussed in part 3.1. In 
relation to each cluster, the way students responded to sets of questionnaire items are reviewed. At each 
level, descriptive statistics and the results of the Chi-square analysis are used. Results from the log-linear 
analysis and the hierarchical log-linear analysis are discussed separately and in view of the specific 
objectives of these analyses. In appendix I the reader can find the questionnaire with an overview of the 
student responses. The proportions reported in the tables are those resulting from the weighting procedure. 

4. 1 Prior knowledge state tests 

4.1.1 Students have no clear view on their prior knowledge. 

In relation to this hypothesis four items have been incorporated in the questionnaire: 

a Did earlier Ou -courses you studied, build on wrong expectations about your prior knowledge level? 

(variable = pkall). 

The results reveal that in the perception of most students (83.2%), the Ou-courses in general do not build on 
inaccurate expectations about ones prior knowledge level. This picture is not different for specific student 
types when taking account of prior educational level or the content domain they study. But, when comparing 
students with different degrees of study success in distance education, a significant difference emerges 
(X 2 =42.829 p. 0000). It seems that especially the group of students who did not already obtain a Ou- 
certificate or only two certificates (51.3% of the respondents) indicates that they have no problem (90.8% to 
90.2%). There is no independency between these variables. When students are more successful (3 to 4 
certificttes or more Ou-certificates) the proportion of students indicating they have no problem with prior 
knowledge drops to 79. 1 % and 73.4%. 

b The course you actually study, does it build on prior knowledge you don't possess ? (variable = 

pkthis). 

The results reveal that in the perception of most students (88.2%), the Ou-course they are actually studying 
does not build on inaccurate expectations about ones prior knowledge level. 

A test of independence with each specific student type variable reveals that there might be an interaction 
with the content domain students study (x 2 "23. 872 p. 0006). The proportion of students indicating they have 
prior knowledge problems is extremely different when comparing natural science and business science 
students (23.5% compared to 8.4%). 

c - What part of the course you actually study did you already master, taking account of your prior 

knowledge, when you started studying? (variable name— pkbegin) 

- The impression about your initial mastery, based on your prior knowledge, could change when 
you started to study the course in detail. What part of the course did you finally estimate to master 
when you initiated your study of the actual course you have taken? (variable name— pkafter) 

In this section we discuss the responses to two questionnaire items. Table 2 shows the proportions of the 
course the students estimate they already master as part of their prior knowledge at two different moments 
of the study process: before and during studying. The estimations of prior knowledge levels does not seem to 
differ depending on the two different moments. The following table repeats this information but reports 
more accurate data: 
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Table 2: Estimation of prior knowledge before and during study 



0-9% 10-24% 25-49% 50-74% 75-89% 90-100% 
before the study 33.8% 36 J % 20.3% 6.3% 1.3% 0.2% 
during the study 34.9% 38.2% 19.1% 6.4% 1.2% 0.2% 



When comparing the data in table 1, we hardly perceive differences. A x*-test confirms that there are no 
significant differences in proportions (x 2 =.2354). 

The comparison and the lack of differences questions the competence of the students to estimate their own 
prior knowledge state level. Next to comparing the outcomes at two estimation moments, one can ask 
whether the percentages in the table are not in conflict with the results obtained from earlier questions 
(section 4. 1. 1 b). For instance, up to 88.2% of the students indicate that while studying a specific course, 
the courses do not build on wrong expectations about one's prior knowledge. Once more these findings 
support earlier research results by Dochy (1992) sustaining the generally accepted statement that students 
are not able to estimate their own prior knowledge level (Falchikov & Boud, 1989). 

4.1.2 Students do not need prior knowledge state tests 

a I would like to get the opportunity to measure my prior knowledge state level before starting a 

Ou-course (variable = wannest). 
This first question, helping to answer the above statement, already presents interesting results. At first 
sight, the proportions of students agreeing, disagreeing or reacting neutrally to the question are not 
extremely different: 35.5% of the students explicitly indicate they wish to know more about their prior 
knowledge state; 33.7% express a neutral attitude and 30.8% express a negative attitude. But, when we 
explore potential interrelations with student type variables, we discover significant differences for each of 
the three variables : educational level (x 2 = 11.563 p.0030); study success (x 2 = 117.37 p. 0000); study 
domain (x 2 40.285 p.0001). 

Some striking differences were found. The number of students expressing a neuU'al or positive attitude is 
higher when the students have no higher educational background. The number of students in favour of 
prior knowledge state tests is extremely higher when we compare students without a Ou-certificate 
(44.0%), only up to two (32.9%) and more than 4 certificates (16.2%). Moreover, 50.2% of the latter 
group explicitly state that they do not want such tests. 

Also, remarkable differences were found in student reactions depending on the study domain of students. 
Especially students studying natural (50. %) and social (40.6%) sciences, express a need for prior 
knowledge state tests. Students studying law (35.5%) and economical (35.1%) express especially a 
negative attitude. 

b How much time would you be willing to invest in prior knowledge state testing? (variable = 

pktime). 

Students studying in a distance education setting are sensitive to their time management and interferences 
in their time schedule. The time constraints they have to cope with in their particular study setting are not 
to be underestimated. It is therefore relevant and necessary to ask students about their willingness to invest 
time in services offered by the institution. 

The latter is already obvious when we see that up to 29.4% of the students indicate right from the 
beginning that they do not want to spend time on solving a prior knowledge state test. This is in 
congruence with the proportion of students indicating they are not interested (30.8%) in prior knowledge 
state tests in the former section. For the further analyses we excluded this subgroup of students from the 
analysis. 

The results show that the willingness to invest time is in general high if the test would last less than an 
hour and a half (59.6%). However, this pattern differs significantly if we take account of the student type 
variables. The proportion of students without a higher education experience that is willing to invest even 
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more th*n half an hour is significantly higher (19.5% compared to 10.9 %). The inverse is also true, 
students with a higher education background prefer to invest less time ( 66.3% compared to 60%). This is 
shown in Figure 6. 



[insert here Figure 6] 

Figure 6: Time investment in prior knowledge state tests; interaction with educational level 

Also the degree of study success influences the pattern in student responses. If we compare students 
without a certificate with those with more than 4 certificates the pattern differences are evident. Among 
the former group the highest percentage is willing to spend between 30 and 90 minutes (67.1%). Among 
the latter 78.8% of the students are ready to invest up to 60 minutes for a test. This suggests that 
'successful' students express a will to manage more strictly their available study time. 

If we compare students according to the content domain they study, meaningful differences can be 
observed. It is for instance interesting to compare economics and natural science studen!s in Figure 7. The 
highest proportion (40%) of economics students only wants to invest up to 30 minutes. The highest 
proportions of natural science students prefer an investment between 30 and 60 minutes. Striking is also 
that 28% of these students are willing to invest more than 90 minutes. 



[Insert here Figure 7] 

Figure 7: Time investment in prior knowledge state tests; interaction with study domains 

c The information obtained during prior knowledge state testing could be represented with 'prior 

knowledge profiles' that reveal your weaknesses and strengths. What kind of use would you make 
of this information? 

From the results of prior knowledge state tests, as stated earlier, profiles can be derived that reveal the 
weak and strong points in a students' prior knowledge. The general question above introduced, contains a 
list of ways the information from knowledge profiles could be used. In a way, the responses to this set of 
'information uses' are helpful to determine whether the informative value of knowledge profiles can be 
considered as specific or as general. 

Nine 'information uses' were put forward in the questionnaire. These nine can be clustered into three 
particular sets : 

set 1: to direct content choices in relation to the study process (What do I already master about 

a course? What subparts of the course will I study more profoundly?); 

set 2: to direct planning behaviour in relation to the study process (What course will I choose? 

How will I plan my study? What will be my study route? I will plan accordingly my 
examination in an earlier phase. I will study in a more rigid way.); 

set 3: to ask for help, su p p ort (For what specific subparts will I call for help? To direct my 

demands for support.). 

Figure 8 gives a general overview of the proportions of students in favour of these three sets of 
information use. 

[Insert here Figure 8] 

Figure 8: Informative value of prior knowledge profiles; % acceptance of three cluster. 

Asking students to react to the use of prior knowledge profiles to direct content choices (the first set) we 
observe high proportions of students preferring this type of information use: 
- What do I already master of a course (variable = in/what)! (51.4%) 
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- What subparts of the course will I study more profoundly (variable = inftopicsp (42.0%) 
In relation to the first question "What do I already master of a course?", the answer pattern is significantly 
different considering all student type variab'es. Students with a higher education background, students with 
no Ou-certificate and students in specific content domains (business, economical, natural, social and 
technical sciences) especially stress this type of knowledge profile use. 

Considering the second question "What subparts of the course will I study more profoundly?", the 
independency hypothesis is ?.iso rejected for the variables study success and study domain. There are for 
instance high proportions of students without a certificate (47.3%) and students sti-c'ying natural sciences 
(55.8%) in favour of this information use. 

The use of knowledge profiles to plan the study (second set) is also acknowledged by the students, but 
with varying degrees of enthusiasm (tabel 3). 



Table 3: The use of knowledge profiles to plan the study 



What course will I choose (variable = inf choice)! 48.0% 

How will I plan my study (variable = injplanp 40.9% 

What will be my study route (variable = infroutep 37.5% 
I will plan accordingly my examination in an earlier 

phase (variable = infexam). 28.0% 

I will study in a more rigid way (variable = in/rigid), 26.5% 



What is striking in this overview is that from the moment the type of information use affects a certain 
level of freedom and independence of the student, the students do no longer agree. From table 1 we can 
also derive that there are significant interactions with student type variables. The degree of success in 
studying interacts e.g. with the first question "What course will I study?". Students without a certificate 
agree with this type (55.2%) of information use. Students with more than 4 certificates strongly disagree 
with this kind of information use (71.6%). 

The student type variable 'content domain* influences the response patterns for 4 of the five questions. We 
summarize this in table 4. This table shows for each subquestion the two groups with the highest 
proportions in favour of or pro a specific information use. 



, Table 4: Content domains and planning based on knowledge profiles 

pro 

What course will 1 choose? T:4L9% L:43.8% E:57.2% N;60. 7% 

How will I plan my study? L:32.6% C:33.6% S:46. 1 % N:57.4% 

1 will plan accordingly my 

examination in an earlier phase. C:22% B:23.8% E;32% T:32.6% 

I will study in a more rigid way. C:18.5% N:21. 7% E:32% T:32.2% 



T= Technical Sciences; N= Natural Sciences L* Law; E= Economic* C- Cultural Sciencei; B« Business 
Studies; S* Social Sciences 



The use of knowledge profiles as a base to ask support/help (third set) is less preferred by the students. 
Only smaller proportions of student favour this kind of knowledge profile use: 
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Table 5: The use of knowledge profiles to ask for support/help 

For what specific subparts will I call for help (variable = in/help/} 25.4% help 1 

To direct my demands for support (variable = in/support). 25.6% help 2 

Moreover, and here there are striking differences, in relation to all student type variables there are 
significant interactions. Students with a higher educational background are e.g. less enthusiastic for the 
information use 'help' (18.0%) or 'support' (20.1%), compared to students with a lower educational 
background: 'help' (32.9%), 'support' (31.9%). 

The proportions of students in favour of these kinds of knowledge profile use are also significantly 
different when we consider the independent variable 'study success'. We compare e.g. the proportions of 
students who respond positively according to the number of Ou-certificates tht students already gained. 
The more certificates acquired, the less enthusiasm can be perceived for help- and/or support-functions. 

Table 6: The use of knowledge profiles to ask for help for different students 

0 cert. 1-2 cert. 3-4 cert. > 4 cert, 
asking for help 29.5% 25.1% 17.7% 16.1% 

asking for support 28.4% 24.9% 18.2% 21.4% 

Considering the third independent variable 'content domain', there are also striking differences in 
proportions in favour or against this kind of knowledge profile use. If we simply consider the data in the 
next table we see that consistently the same content domains display the highest pro reactions: 

Table 7: The use of knowledge profiles to ask for help in different domains 
pro 

asking for help B.17.2% T:23% E:35.4% S:36.4% 

asking for support B:l 7% T:26. 1 % S:31 . 7% E:36. 6% 

T= Technical Sciences; N= Natural Sciences; L= Law; E= Economics; C= Cultural Sciences; B= Business 
Studies; S— Social Sciences 



d Personalised and written feedback in relation to self-tests should be a valuable option, (variable - 

selftest) 

The potential of prior knowledge profiles is that students can get personal or written feedback. 63.8% of 
the students indicate they would appreciate this kind of feedback. Only 11.5% is not interested in this 
provision. Further analysis reveals that certain student type variables influence the response pattern of the 
students. The number of students depreciating this provision is for instance higher when the students have 
a higher education background (15.7% compared to 6.5%). 

The same pattern can be found if we compare students having as yet no Ou-certificate (6.0%) and students 
who already obtained more than 4 certificates. 28.3% of the latter group does not like this extra provision. 
Considering the content domain the students are studying, there are significant differences in the 
proportions of students agreeing or disagreeing, but the overall number of students showing a positive 
attitude is for all content domains more than 57.6% with a maximum of 68.7% (social sciences). 
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e When taking examination at the Ou, I learn very little from the feedback or information in 

relation to my errors, (variable = exam). 
This item is not immediately linked to the topic 'prior knowledge state tests*, but - in our perception - 
results in relation with this item bring further evidence for a testing practice, resulting in more relevant 
informative feedback. Moreover, the topic is linked to the 'feedback'-issue tackled in the former section. 
These students could agree, disagree or react neutrally to the following statement: 
"I learn little from my final examination. I get too little feedback or information in relation to wrong 
answers or solutions." The students react in the following way (N= 1839) (table 8). 



Table 8: Learning very little from the feedback on examinations 



/ disagree 
neutral 
I agree 



16.8% 
44. 7% 
38.4% 



From Table 1 we can derive that student reactions interact with all student type variables (x 2 ). 
For instance, students without a higher education background (41.5% agree, 13.8% disagree) express in a 
more explicit way their agreement than students with a higher education background (36.9% agree, 18.8% 
disagree). Study success as expected interacts extremely with students opinion about the feedback value of 
current Ou-examination. 

Table 9: Appreciation of the current feedback practice ; interaction with the number of certificates 



number of certificates % agreement % disagreement 



0 


23.6 


12.8 


1-2 


58.1 


16.3 


3-4 


54.8 


22.9 


>4 


57. .2 


26.3 



Also the interaction with the content domain the students study is significant; cf. Figure 22. Disagreement is 
the highest within students studying technical (24.9%), economical (21.0%) and law sciences (20.0%). 
Agreement is u '\gh within students studying law (44.5%), social (43.5%) and business sciences (40.7%). 



4.1.3 The perceived and expected effects of prior knowledge measurement on study variables are limited 
a Prior knowledge has an effect on 'selectivity* in study behaviour. 

Students could, in relation to this hypothesis, react upon two questionnaire items. The first item builds on the 
condition where students themselves estimate their own prior knowledge level. The second item builds on 
information that could be provided to the students as a result of prior knowledge state tests and prior 
knowledge profiles. 



Table 10: Prior knowledge and selectivity in study behaviour 

/ agree neutral I disagree 

I concentrate on course parts that are new to me 39.3% 39. 7% 21 A % 

(variable = pkselect) 

1 would select course parts to he studied more 46. s% 30.8% 22.8% 1 
thoroughly for the final examination 

(variable = choice) 
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The percentages suggest that prior knowledge measurement could influence selectivity in study behaviour. 
In relation to the first statement, student reactions interact with two independent variables 'educational level' 
and 'study domain'. The proportion of students concentrating on new parts as a consequence of estimated prior 
knowledge state levels, is higher when the students have a higher education background. Especially students 
studying economical sciences (51, 9%) and technical sciences (51, 7%) agree with this statement. 

But reactions to the second statement are also interrelated with student variables. Striking is for instance the 
difference in proportions of students agreeing when we distinguish students without certificates (50.6%) versus 
students with more than 4 certificates (34.8%). 

Natural sciences (29.5%) and business sciences students (28.5%) react most negatively, whereas the 
proportionsof students with a positive attitude are the highest among technical sciences (48.2%) social sciences 
(47.4%) and law sciences (47.5%) students. 

b The information acquired from prior knowledge state tests can influence study time management. 

Two questions in the questionnaire asked whether students would change their study speed when they got 
information about their prior knowledge state level. 



Table 11: Prior knowledge state tests and study time management 



/ agree neutral I disagree 
I would take examination earlier 34. 1% 39.6% 26.3% 

(variable = quicker) 

Yes No 

I would plan in a more rigid way my study 26.5% 73.5% 

(variable = in/rigid) 



The data in the table are interesting. At first sight a contradiction in the response behaviour of the student is 
suggested by the very different proportions of students reacting negatively. But, both statements ask students 
to react to different aspects of study time management: 

the first statement leaves all autonomy to the student to determine the moment to take an examination. A 
certain proportion of students would be willing to do this (34. 1%). 

The second statement asks for an attitude towards a more drastic, long-term aspect of time management. Here 
we perceive an extremely negative attitude towards such kind of time management planning. 73.5% of the 
students reject this option. 

The data in the table get even more interesting when we research significant interactions with student type 
variables. Study success and study domain influence responses to the first statement. Experienced students with 
more than 4 certificates (33.9%) react e.g. more negatively than students without a certificate (24.5%). 

In contrast, the proportions switch when we look at the proportions of students in favour of the option offered: 
the proportion of experienced students drops to 24.2% but the proportion of students without certificate raises 
to 36.6 %. Students studying social sciences (37.3%) and technical sciences (43.2%) are more positive towards 
this type of time management whereas law students (31.7%) and economics students (42.1%) express the 
highest proportions of negative attitudes. 

In relation to the second statement there is only a significant interaction with the study domain: cultural 
sciences (81.5%) students and natural sciences (79.3%) students react much more negatively. Technical 
sciences (32.3%) and economics sciences (30.2%) students react more positively. 

c. Considering the potential of the technological evolution, it is possible to present individual students 

a specific printed version of a course. Such an individualised print-out, matched to my prior 
knowledge and experience, is a good option, (variable name* print) 
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If recent technological developments were us i to derive from prior knowledge measurement a 'tailored 
course* by printing an adapted version of the course, students do not express an extremely negative (36.5 %) ; 
neutral (33.0%) or positive attitude (30.5%). 

But this global picture changes when we perceive significant interrelations between response behaviour and 
all the student type variables (table 12). 



Table 12: Students' perceptions on using inidividual printed courses matched to their prior 
knowledge 



No higher education background 
Higher education background 

No certificate 
> 4 certificates 

Business sciences 
Natural sciences 



I disagree I agree 
32.6% 34.3% 
41.7% 26.4% 

32. 7% 35. 1 % 
45.9% 21.0% 

42.7% 26.8% 
32.4% 37.0% 



The first two variables indicate that experienced students reject more readily this possibility. 

d. Students with a good portion of prior knowledge in a certain content domain should get the 
opportunity to follow a suited study route, (variable name= route) 

At this moment students, taking courses from a diploma line have to take a standard number and/or elaborated 
set of courses. When asked to react to this statement, we can measure to what extent students wish to get 
adapted study routes (e.g. shorter). 

[Insert here figure 9] 

Figure 9: Prior knowledge state and study route; interaction with study domain 

The proportion of students in favour of this possibility is very high (62.4%). And in line with expectations 
there are significant interactions with student type variables: study success and study domain. Although more 
than 50% of all students is in favour of this option, we see that students without a certificate show rather 
extreme positive reactions (66.0%) than students with more than 4 certificates (50.5%). 

When controlling for the domain the students study, as shown in figure 9, we perceive an extremely positive 
attitude of students studying natural sciences (70.5%), social sciences (69.2%) and technical sciences (69,3%). 



e. When studying my course, I need a tool to determine 
course-content, (variable name— mastery) 



on my own - how good I already master the 



72.5 % of the students accept this statement. Only 7.7% reject it. This attitude clearly indicates the need of 
students to have more information about their study progress. 

The significant interactions in student responses with student type variables do not change this picture. They 
only reveal that e.g. students with no higher education background (76.8%) do express this need to a higher 
extent than those with a higher education background (68.4%). 

A comparable interaction is perceived when controlling for the number of certificates: students without 
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certificate (76.3%) are more positive towards the statement than students with more than 4 certificates 
(60.9%). 

[Insert here Figure 10] 

Figure 10: Prior knowledge tests: a tool to control mastery of the study process; interaction with study 
domains 

The significant interaction with study domain is especially due to the different proportions in favour of the 
statement among natural sciences students (79.6%) and economics sciences students (65.5%). 

4.1.4 The perceived effect of PKST on guidance, support is limited 

Three questions in the questionnaire measured the extent the students were willing to accept guidance or 
support based on the results of prior knowledge state measurement. We summarize the results in table 13. 

Table 13: Perceived effect of prior knowledge state tests 

No Yes 

1. I would use information on my prior knowledge 
to direct my demand for support (variable = infsupport). 74,4% 

2. / would use information on my prior knowledge 
to ask for help from a student tutor (variable = infhelp). 74.6% 

I disagree Neutral I agree 

3. / would use the information to ask advise from 
experts in relation to the parts I have to study 35,6% 32 A % 32 J % 

(variable = advise) 

Making use of a third party to direct one's own study does net seem to be an attractive option to most 
students. The high percentages of 'no* reaction support this conclusion. 

From Table 1 we leam that the way students react is also significantly influenced by the three independent 
variables: educational level, study success and study domain. The following 2 figures depict these interactions. 

[Insert here Figure 11] 

Figure 11: Prior knowledge and help/support; interaction with student type variables 
[Insert here Figure 12] 

Figure 12: Prior knowledge state tests; advise from experts, interaction with student type variables 



For example, the proportion of students in favour of the three statements is always higher among students 
without a higher education background than among students with this background. Secondly, when comparing 
students without a certificate and those v ith more than four such certificates the same consistent picture can 
be perceived: the proportion of students without a certificate in favour of the statement is always higher than 
the proportion of students with more than 4 certificates in favour of the idea. Thirdly, checking for the 
variable * study domain*, reveals that the proportions of students taking courses in economics sciences and 
social sciences, which are in favour of the idea,are much higher than the group of favourable students who 
follow courses in other content domains. 
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4.1.5 Log-linear results 

As stated in section 3.4, we incorporated in our analysis techniques log-linear and h ;rarchical log-linear 
analyses for a specific purpose. In contrast to the x 2 "technique we try to check to what extent two of the three 
independent variables also play a combined effect to explain the specific data-structure in the three-dimensional 
table. We repeat that: 

- our log-linear analysis focuses on a test of independence of the three variables in the analysis and takes 
account of the first two student type variables as independent variables and the specific answer on a 
questionnaire item as the dependent variable; 

- the hierarchical log-linear analysis explores what kind of model can explain the data in the three-dimensional 
table. This model is expected to be the simplest model explaining still the highest proportion of variance. 

Table 1 summarizes the results of these analyses (G 2 and p-values). We focus in this section only on the 
analyses in relation to prior knowledge state tests. The test of independence of the three variables in the model 
is highly informative. In relation to most dependent variables about prior knowledge tests we perceive that the 
independency hypothesis is to be rejected (p < .01). This confirms the results of the two-dimensional ^-tests. 
Example: analysis of the results for the question whether students want prior knowledge state tests (variable = 
wannest) reveal a very significant G 2 -va!ue (p=5.E-21). We can read in the three first columns, the very 
significant results of the chi-square analyses; each independent variable interacts with student response 
behaviour. The log-linear analysis adds to this that it might be possible that a combined effect of the two 
independent variables * study success 1 and 'educational level' influences the response pattern. Very significant 
p-values are indicative of this possible phenomenon. The hierarchical log-linear analysis confirms this - to a 
high extent - since the final model contains the interaction between wanttest x educational level £nd the 
interaction between wanttest x study success. This finding is a first indication that the independent variables 
'educational level' and 'study success' are useful operational definitions of the variable 'student type'. Of high 
importance is the fact that the interaction of both independent variables (educational level x study success) is 
not a part of the final model. 

The table also shows three examples of models containing third-order interactions: the variables pkthis, pkall 
and pkiifter. In these three cases, the final model contains also the third-order interaction between tlie two 
independent variables (IV , and IV^) and the dependent variable (DV) £nd all second-order interactions ind 
main effects of the single variables: 

(IV, x IV, x DV) + (IV, x DV) + (IV 2 x DV) + (IV, x IVJ + IV, + IV 2 + DV. 

This finding does indicate that for these three dependent variables the impact of each single independent 
variable or their interaction with the DV is not enough to account for high proportions of the variance. 
But, it is remarkable that the^e three dependent variables are linked to the first questions in the 
questionnaire. This could suggest that students were still orienting themselves towards the topic 'prior 
knowledge state tests' when responding to these items. The impact of background variables could be 
overruled by this 'orientation behaviour'. 

In relation to three variables the independency hypothesis is not rejected for the log-linear analysis: 
infplan, in/exam and infrigid. Here we perceive in the final model, after a hierarchical log-linear analysis, 
that only main effects of the variables in the model are retained to account for the highest proportion of 
variance. The single effect of each independent variable is always retained in the final model, thus giving 
extra support to our earlier statement that these IV can be considered as good operational definitions of the 
general variable 'student type'. 

In general, the results of the log-linear and hierarchical log-linear analysis confirm the results obtained 
from the x 2 -analyses. But they add an extra dimension to these results by indicating that the discrete 
independent variables 'educational level' and 'study success' are each but part of a complex set of possible 
operationalisations of the concept 'student type'. Two-way interactions and even three-way interactions 
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also play a part. But, taking account of the high level of agreement between the ^-results and the results 
form the log-linear analyses, the results indicate that the two independent variables in our models are 
already useful and relevant operational conceptions of the complex variable 'student type'. 



4,2 Progress tests do not answer specific student needs 

This hypothesis about progress tests has been spilt up into two subhypotheses: 



4.2. 1 The possibility for students to determine study progress by making use of progress tests is an 
option to be offered, (variable name= wish) 



Table 14: Using progress tests 



I agree 75.3% 
I disagree 7.8% 
Neutral 16.9% 



The results in the table 14 hardly need further comments. This could be predicted from the way students 
reacted to the statement in section c.5. There is also a significant interaction with each of the three student 
type variables. The difference between the proportion of student in favour of progress tests with higher 
education (70.4%) and without higher education (79.5%) is already remarkable. But, if we compare 
proportions of students in favour of progress tests without a certificate (83.0%) and with more than 4 
certificates (55.2%) the difference becomes even more striking. 

Thirdly, the content domain the students study also plays a role: Business sciences students (10.3%) and 
law sciences students (10.6%) are much more against the idea of progress tests than students studying 
other domains, whereas the proportion of students in favour of the idea and studying natural sciences 
(88.6%) is up to 10% higher than students in other domains. 



4.2.2 To get regularly the possibility to take progress tests would be helpful to structure my study in 
time and to take examination much earlier, (variable name- regular) 

Only one question of the questionnaire evaluated the potential impact of progress test measurement on 
study behaviour. With 55.7% pro and 17.1 % against the idea of making use of progress tests to adapt 
ones study time management, progress seem to answer a certain need of students. 

That this 'need* is different between students can be derived from the chi-square results. Students without 
higher education background (79.5%) are more positive towards the idea than students with a higher 
education background (70.4%). If we compare proportions of students in favour of but without a 
certificate (67.6%) or with more than 4 certificates (33.0%) the difference becomes even more striking. 

Students taking courses in cultural sciences (58.1%) and social sciences (58.7%) are to a higher extent in 
favour of time management based on progress tests than students studying other domains. 

4.2.3 Log-linear results 

The discussion presented in 4.1.5 can be repeated in relation to student opinions and reactions in relation 
to progress tests: 

Firstly, the results support the findings of the ^-analyses. The independency hypothesis is to be rejected. 
There is an interaction between the student responses to progress tests and the two independent variables 
'educational level* and 'study success'. 
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Secondly, the analyses confirm that the independent variables are useful operational definitions of the 
general variable 'student type'. Both variables wish, atxd regular, in interaction with the independent 
variables, are retained in the final model after hierarchical log-linear analysis. No three-way interactions 
are observed. 



4.3 Study contracts do not answer specific student needs. 

As stated in section 3.1, study contracts do not represent an alternative evaluative format. We have 
incorporated this subject as a logical extension to the alternative evaluation formats 'prior knowledge state 
tests' and 'progress tests' in order to get extra information on the students' views about the consequences 
they are willing to attach to the results of new evaluation formats. Actually, at the Dutch Open university, 
there is a vivid interest in study contracts as a way to strengthen the institute-student link, assuming a 
better and higher degree of students taking follow-up courses (Van der Velden, et al. f 1992). Part of the 
potential content of a study contract could be related to (formal) use of evaluation results. 

4.3.1 After getting info about my prior knowledge and advices based on it, I would be prepared to 
discuss with the to make a 'contract' about my study route and study planning, (variable name = 
contract) 

32.7 % of the students accept the idea of 'study contracts', 45.4% reject the idea and 22.0% express a 
neutral attitude. Acceptance or rejectance is significantly influenced by the three student type variables. 
Figure 40 depicts the impact of 'educational level' and 'study success'. The negative attitude is for 
instance much higher among students with a higher education background (52.5%) than among students 
without this educational experience (39.0%). 

Also the variable study success plays a comparable role: students without certificates are much mo r e in 
favour (35.9%) or less against (41.5%) than those with already more than 4 certificates (29.3% in favour, 
53.5% against). 

As to the variable 'study domain', students studying business sciences (52,5%) and economical sciences 
(49.0%) are much more against study contracts than those taking courses in other domains: 

The proportion of students in favour of this potential initiative is higher among the law sciences (33.2%), 
social sciences (39.9%) and technical sciences (33.2%) compared to students studying other content 
domains. 

4.3.2 Log-linear analysis 

Again, the discussion presented in 4.1.5 can be repeated here as we already did in section 4.2.3: 
First of all, the log-linear analysis results support the findings of the ^-analysis. The independency 
hypothesis is rejected. There is an interaction between the student responses to study contracts and the two 
independent variables Educational level' and 'study success'. 

Second, the analysis confirms that the independent variables are useful operational definitions of the 
general variable 'student type'. Both variables 'wish' and 'regular', in interaction with the independent 
variables, are retained in the final model after hierarchical log-linear analysis. No three-way interactions 
are observed. 
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5 Discussion 

In discussing the results sported above, we will structure the discussion according to the six sets of 
hypotheses as stated earlier. But, considering some particularities of the response behaviour of the 
students, we start the discussion with some general reflections. 

5.1 Responding to the questionnaire: a learning process 

When examining the construction of our questionnaire we stressed complicating issues that could interfere 
with the response behaviour of the students since they are asked to react on evaluation formats they are not 
acquainted with. The solution to this problem was a careful construction of the questionnaire by gradually 
presenting to the students very specific potential implications/consequences of alternative evaluation 
formats. This brings about that the students gradually are introduced "to a fuller picture of these alternative 
evaluation approaches and that their initial responses to questions can be - from this point of view - 
different from those at the end of the questionnaire. We could, if the latter is true, for instance expect that 
the students take more and more extreme positions towards the items presented at the beginning or end of 
the questionnaire. A quick glance at the response patterns in the tables of appendix I shows that this is not 
the case. The opinions vary, the proportions of neutral, positive or negative responses switch from item to 
item. Moreover, and this will be illustrated in the next sections, the response pattern is rather consistent 
when comparable implications/consequences are presented to the students. 

Nevertheless, a certain 'learning process* could yet be observed when analyzing the students responses. 
We already referred to it in dealing with the log-linear results about prior knowledge state tests. There we 
found three-way interactions between the dependent and independent variables to be able to account for 
variance in the three-dimensional matrix, and this in relation to some initial items in the questionnaire. We 
tried to explain this by suggesting that the students were still orienting themselves towards the topic 'prior 
knowledge state tests' when responding to these items. The impact of background variables could be 
overruled by this "orientation behaviour*. 

5.2 Prior knowledge state tests 

Four main hypotheses were put forward in relation to prior knowledge state testing. The results of each of 
these will be discussed in relation to the theoretical background and earlier empirical findings. 

5.2.1 Students have no clear view of their prior knowledge level. 

The information in the results section, based on the reactions of student to three items, is helpful to accept 
or reject this hypothesis. 

First of all it is remarkable that in general, students indicate no major problems in relation to courses wich 
would build on wrong expectations about their prior knowledge. This could suggest that students are not 
aware of mismatches between course content and their prior experiences/knowledge. 
But this is countered by various tests of independence with independent variables. We perceive for instance 
a very logical and coherent pattern in the way students who already acquired more certificates react to 
these items. These more experienced students take second-level and/or third level courses and enter 
domains for which they have less prior knowledge. As a result it is logical that they mention higher 
degrees of mismatches between their prior knowledge state level and the course content elaboration. 
Also interesting in this context is the fact that students taking courses in study domains that build on very 
explicit prior knowledge mention proportionally higher degrees of prior knowledge problems; a typical 
example is the study domain natural sciences. The coherence and nature of the findings discussed here 
give us not enough grounds to reject this first hypothesis in relation to prior knowledge state tests. 
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The fact that past research has shown that students are not able to rate their own level of prior knowledge 
(Falchikow & Boud, 1989; Wagernans, et al., 1991; Dochy, 1992) can be sustained. Moreover, an 
important finding is that general aspects such as the domain of study provides interesting information. This 
investigation shows several times that prior knowledge and its use is domain-specific (Alexander, 1992; 
Dochy, 1992). 

5.2.2 Students do not need prior knowledge state tests. 

Five items in the questionnaire were presented to the students to be able to test this hypothesis. Prior 
knowledge testing was presented as affecting the following study behaviour and study process aspects: 

- uncertainty about one mastery when starting a course 

- time management 

- the way information obtained from prior knowledge profiles is to be used 

- the need for personalised and written feedback 

- the lack of feedback in current evaluation practices 

Generally, it can be concluded that there is a need for prior knowledge state testing, i.e. 69.2% of all 
respondents don't have any objections. Interesting is that this need is concentrated within certain 
subsamples, being students with lower educational background and students without and with no more than 
two certificates. Also interesting is that the nature of the domain plays a role in this: for example 50% of 
all students in natural sciences is in favour. 

Striking is that most students in favour of prior knowledge testing do prefer to spend considerable time on 
this: one hour and a half. This time management should be seen in the light of prior educational 
experience: students with higher education experience do want to spend less, those without higher 
education experience want to spend more time. Also, the less certificates a students has, the more time he 
wants to invest. 

Concerning the use of information obtained from knowledge profiles, three types of use were selected: to 
direct content choices, to plan the study process, to ask for help and support. 

Half of the students do want to know what they already master from a particular course. Students with a 
higher education background, students with no certificate and students studying business studies, 
economics, natural, social and technical sciences especially stress this type of knowledge profile use. 
Knowledge profile use in order to plan the study is not that popular, except for choosing courses (48%) 
and for students with no certificates (55.2%). 

Knowledge profile use as a base for support is also particularly requested by students with no higher 
education experience and without any certificates. 

An advantage of the use of knowledge profiles is that students receive personalised and written feedback, a 
service that is highly welcomed by students. Only 11.5% of the students is not interested in this provision, 
being students with a higher education background and with several certificates. Students generally find 
that there is a lack of feedback when talking about the current examinations. The more exams a student 
takes, the more need for specific feedback. 

Apart from the view and utility on the different functions of knowledge profiles, the variables that are of 
prime importance are stressed. As shown earlier by Dochy & Steenbakkers (1988) and Dochy (1992), 
previous educational level operationalised as having 'higher education' or 'no higher education' is a 
significant variable. Next, as earlier shown by Alexander (1990) and many others (for an overview, see 
Dochy, 1992) using prior knowledge is domain-specific. The variable 'domain' seems to be of importance. 
Finally, study success expressed as an amount of certificates is significant as found earlier by Powell et al. 



(1990). 
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5.2.3 The perceived and expected effects of PKST on study variables are limited. 

The results show evidence to reject this hypothesis. Related to our second statement related to information 
that could be provided to students as a result of PKST, 77.2 % of the students say not to disagree with the 
fact that they would select parts of the course to be studied more thoroughly. This would even be used 
more by students with a higher education background. 

Related to their time management, as expected on the earlier stated literature, students do not want an 
intervention in their long-term time-management. However, when the autonomy is left to the student, he 
might use a provision (such as PKST) in order to take an examination earlier (73.7%). Also this fact 
differs for domains and amount of certificates. 

Finally, a tailored course matched to one's prior knowledge is acceptable for 69.5% of the students, being 
mainly students with no higher education background and no certificates yet. 
Students are also in favour of specific study-routes when having high prior knowledge. 

5.2.4 The perceived effect of PKST on guidance, support is limited. 

This hypothesis can be confirmed. A larger proportion of the sample does not express a positive answer to 
these questions. This is also in line with earlier research (Powell, Dochy) which shows that involving 'a 
third party' for support is not well appreciated by students. The 2,0% up to 32 % of the sample in favour 
of a third person for guidance are clearly also those students having no higher education experience and 
having no certificates up till now. 

These results are consistent with earlier research. Boon et al (1990, p. 31) discovered e.g. that 70% of a 
sample of students (N= 1123) prefer the actual educational system with much freedom or even more 
freedom instead of a system with restrictive aspects. Important in their results was the fact that students 
sometimes expect more provisions, but objected against shortcutting of their freedom as a consequence of 
these provisions (p. 30). 

5.3 Progress tests and study contracts 

a Progress tests do not answer specific student needs. 

In answer to the question if students need a tool to determine themselves how good they master the 
content, 92.3% replies that it is the case. This clearly shows an extremely positive attitude towards 
progress tests and confirms the findings of earlier research as given in the introductory part. Thsi few 
students that do not want this are in the camp of higher education background and more than 4 certificates. 
A second, but direct question concerning the use of progress tests reveals exactly the same results (7.8% 
against). Differences between several groups are rather small, except for differences in domain (deviation 



Apart from this positive result, with 55.7% pro and only 17.1% against, the idea of using progress tests to 
adapt ones study time management seels to answer a real need of students. 

Since progress is shown to be one of the major problems in distance education (e.g. Powell, 1991; Dochy, 
1992), the implementation of progress tests could probably provide a solution. 



Since it is clear that students cannot estimate their own prior knowledge level, the fact that students do not 
like interventions in their time-management from a third party (this investigation) and are not able to state 
their own study program intentions (Powell, 1991), it is expected that this hypothesis cannot be rejected. 
This seems to be the case. In congruence with the finding of Van der Velden et al.(1992) that only 16% of 
the students is interested in study contracts, we found 32.7% of the students accept the idea, but 45.4% 



of 10%). 
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Study contracts do not answer specific student needs. 
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reject it. Striking is that students in business sciences and economics are even more against study 
contracts. 



6 Conclusions 

This investigation stresses the findings of earlier research and shows that development of concrete 
instruments can lead to a considerable use by the students. Overall, one can s&j that there is a realistic 
need for prior knowledge and progress testing and that this will lead to better study results and a higher 
through-flow of students. Prerequisite is that the correct variables are taken into account. 

Students are not able to estimate their own entry level, however there is a need for assessment of prior 
knowledge and a translation into knowledge profiles. For a majority of the students the individual and 
specific feedback from knowledge profiles is highly necessary and students are willing to invest 
considerable time. This information will be used for making content choices, for planing the study process 
and as a basis for support. Hence, several variables have shown to be very useful for differentiating 
between groups and between functions of knowledge profiles. These variables are: previous educational 
level (higher or no higher education), domain and study success (amount of certificates). These results are 
confirmed and corroborated in our log-linear analyses. 

The study shows that students do not want any interference in their time-management (e.g. by tutors) but 
providing them with tools to monitor their entry-level and progress could lead to a faster study-route. 
Students do favour new tools as tailored courses and study-routes, adapted to individual knowledge 
profiles. Extremely high is the expressed need for a tool such as progress tests. These have also a much 
better chance for influencing the pace of studying, rather than study contracts which are much less 
acceptable for students. 

This investigation shows that the development of alternative assessment procedures will be better accepted 
as a basis for study-support (as shown in the earlier teaching-learning model) than traditional human 
support. Moreover, the different functions it can serve for different groups is larger, and when taking 
account of the main variables large groups of students can get a better through-flow. 
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Table I: Overview of Chi-Square analysis and log-linear analyses 
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